Install the snappy codec so the Kafka consumer can read compressed batches - #126

Problem

kafka-python does not negotiate compression — it has to decompress whatever codec the producer chose — and the codec libraries are optional extras it does not pull in. The image installs no snappy support, so against a producer using snappy every fetch fails:

kafka.errors.UnsupportedCodecError: Libraries for snappy compression codec not found

Why this is a crash loop, not a dropped message

UnsupportedCodecError is raised by the consumer iterator while fetching, not by consume(). That puts it outside the per-message error handling added in 16f9dbe, which only wraps consume():

for msg in consumer:          # <-- raised here
    try:
        consume(KafkaMessage(msg))
    except Exception:         # <-- never reached
        ...
    consumer.commit()

The exception escapes the loop, finally closes the consumer, and the process exits. On restart the consumer rejoins the group and re-fetches the same uncommitted batch, failing identically.

Observed in a stage deployment as a crash loop that advanced the consumer group through 13 generations in three hours without ever committing an offset. Authentication, TLS, topic subscription and partition assignment were all healthy — only the decompression failed.

Fix

Adds python3-snappy to docker/install-dependencies.sh.

requirements.in is deliberately untouched: the Dockerfile installs RPMs and then runs pip install --no-deps ., so the runtime image only sees the dnf list. Adding python-snappy to pip would make CI build it from source and require libsnappy-dev.

Scope

Only snappy is added, which is what the producer in question uses. lz4 and zstd remain unavailable and would fail the same way if a producer switched codec — happy to add those here too if preferred.

🤖 Generated with Claude Code